REMARKS 



Claims 1-33 are pending in the application. Claims 1,18 and 24 have 
been amended. Reconsideration of this application is respectfully requested. 

The Office Action rejects claims 1,2,4, 5, 7, 9, 24 and 25 under 35 U.S.C 
103(a) as unpatentable over U.S. Patent No. 5,500,920 to Kupiec, hereafter 
Kupiec in view of U.S Patent No. 6,272,457 to Ford et al., hereafter Ford. 

Kupiec does not, in fact, teach a method for providing collateral 
information based on the content of an information stream as recited in 
independent claims 1 and 24 as amended. Claims 1 and 24 have each been 
amended to clarify that the results of the database queries constitute information 
that is collateral to the presence of events recognized as occurring in the 
information stream. 

Kupiec is concerned with answering spoken queries explicitly formulated 
by the end user through a process that simultaneously locates relevant 
documents and uses word pair co-occurrence statistics to improve the accuracy 
of the automatic speech recognition applied to the spoken query. In Kupiec's 
system, the queries must be explicitly formulated and spoken by the end user. In 
contrast, claims 1 and 24 recite a method for examining an information stream to 
recognize a content thereof (a presence of events) and automatically generating 
database queries based on the recognized events that will retrieve relevant 
information that is collateral to the recognized events. 

Kupiec's stages that conduct "automatic searches" (column 14, lines 15- 
20, column 34, line 29, and column 2, lines 26-27) are either for selecting the 
most likely phonemes during the speech recognition process, or for searching 
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candidate word pairs (called hypotheses by Kupiec) to find the most likely 
transcription of the words. In either case, the searches are conducted to edit the 
phonemes and not to develop collateral information. Accordingly, Kupiec lacks 
the step of automatically generating database queries based on the recognized 
events as recited in claims 1 and 24. 

Moreover, Kupiec does not teach the step of analyzing the database 
results so as to rank and select the results for insertion into the information 
stream as information that is collateral to the recognized events (claim 1) or 
presentation as information that is collateral to the recognized events (claim 24). 
The referenced passages at column 2, lines 26 and 27, and at column 12, lines 
1-6, merely refer to database queries to retrieve documents, which are not 
described as information collateral to the recognized events for insertion into the 
information stream (claim 1) or for presentation (claim 24). Accordingly, Kupiec 
also lacks the analyzing step. 

The Office Action recognizes that Kupiec does not automatically generate 
database queries from the recognized events, but that Ford does, relying on the 
passage at column 2, lines 29-33. However, this passage and the ensuing 
description merely refers to building a data record of corrected location data and 
recognized text data synchronized to a location or a track, the data record being 
stored in a geographic information system (GIS) database and correlated to 
linear networks and point observations. There is no mention of the step of 
automatically generating database queries from the recognized events of the 
examining step as recited in claims 1 and 24. Thus, the combination of Kupiec 
and Ford also lacks the step of automatically generating database queries from 
the recognized events. 
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With respect to claims 4, 5, 7 and 9 and the references to column 2, lines 
45-52, and column 12, lines 34-40, Kupiec's system appears to examine the 
user's spoken question, but does not extract text therefrom. 

Moreover with respect to claim 4, Kupiec's system retrieves documents 
from the database based on a co-occurrence of words in the user's question, but 
not based on information corresponding to a list that identifies topics in text that 
is automatically extracted from the information stream. 

With respect to claim 25, Ford does not teach how to automatically 
analyze these information sources in real-time or near real-time inasmuch as the 
observer's observations are entered into storage for later analysis by a user with 
the interactive tools. The only queries and search processes mentioned in Ford 
have to do with accessing the asset-tracking database after it has been created. 
The only analysis of the input information streams described by Ford is a manual 
process performed during a database record validation and correction phase. 

For the reason set forth above, it is submitted that the rejection of claims 
1, 2, 4, 5, 7, 9, 24 and 25 under 35 U.S.C 103(a) is inapplicable and should be 
withdrawn. 

The Office Action rejects claims 12-17 under 35 U.S.C 103(a) as 
unpatentable over Kupiec in view of U.S Patent No. 5,835,667 to Wactlar et al., 
hereafter Wactlar. 

With respect to claim 12, neither Kupiec nor Wactlar teach a step of 
automatically generating queries from an arbitrary information stream. Rather, 
Kupiec teaches how to formulate a number of searches given an explicit query 
composed by the end user. Similarly, Wactlar only deals with queries explicitly 
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composed by the end user. Also, Kupiec generates multiple search queries 
based on the entire information stream (and not identified information elements 
thereof) that constitutes the search query posed by the user. Thus, Kupiec, even 
in combination with Wactlar, does not teach claim 12. 

Contrary to the position stated in the Office Action, Kupiec does not teach 
a method of providing collateral information for multiplexing with an information 
stream. Rather, Kupiec's system provides documents, which are not 
multiplexed with the original information stream. 

The Office Action concedes that Kupiec does not teach the steps of 
converting, analyzing, extracting and multiplexing, but that Wactlar does. 

Wactlar analyzes text to identify information elements, which are used to 
provide more efficient indexing and to help summarization where the segment to 
be summarized is based on information deduced from the video. Wactlar 
teaches a search task based on a user formulated query (the basic search 
problem), the search being performed based on an index of words that was built 
during the library creation time. Wactlar teaches only of a different display of a 
big hit list, using different types of icons that may show a summary of the video 
result. Wactlar does not teach the step of extracting data from a database 
search result that is relevant to the information stream. Wactlar does not teach 
the step of multiplexing the data into the information stream. Rather, Wactlar 
only teaches different visualizations of the results. 

Regarding claim 13, Wactlar does not teach the step of ranking data 
extracted from the search result since Wactlar does not extract the data and 
Wactlar has no multiplexing step to use the ranked data. 
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Regarding claim 14, Kupiec says nothing about the use of named entities 
(i.e., proper names, people, places, companies, etc.) to improve query results. 
Furthermore, Wactlar does not teach using a taxonomy to segment the 
information stream and using the same taxonomy to classify the user queries. 
Wactlar does not teach reranking the hit list based on the classification (using the 
taxonomy) and the extracted named entities. Wactlar does not teach a taxonomy 
path score. Thus, the combination of Kupiec and Wactlar does not teach claim 
14. 

Claims 15, 16, and 17 recite a method of automatically generating queries 
from an information stream using a topic taxonomy and re-ranking results based 
on named entities. In column 6, line 67 Wactlar describes creating a digital 
library. The creation of the digital library comprises transcribing the audio track 
and indexing (some or all) of the words. The creation of the digital library is 
completely unrelated to the query generation process recited in claims 15-17. 

Wactlar teaches time stamping of the audio stream and the video stream. 
Wactlar teaches aligning these time stamps. Wactlar teaches segmenting the 
video stream (and by correlation the audio stream) into segments based on video 
image characteristics. Wactlar does not teach the step of identifying the topic 
based on a topic taxonomy of the transcribed audio track. 

Wactlar teaches identifying named entities for the purposes of indexing, 
but not for the purposed of reranking the query results, as recited in claim 17. 

For the reason set forth above, it is submitted that the rejection of claims 
12-17 under 35 U.S.C 103(a) is erroneous and should be withdrawn. 
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The Office Action rejects claim 18 under 35 U.S.C 103(a) as unpatentable 
over Kupiec in view of U.S Patent No. 5,970,460 to Bunce et al., hereafter 
Bunce. 

Claim 18 has been amended to clarify that the results of the database 
queries constitute information that is collateral to the presence of events 
recognized as occurring in the information stream. 

The Office Action contends that Kupiec teaches a system for providing 
collateral information for inclusion with an information stream as recited in claim 
18 (the preamble and last paragraph). This contention is untenable because 
Kupiec responds to a spoken search query (the information stream) to provide a 
document that is not inserted or included in the information stream. 

Bunce teaches how low level data that is internal to an automatic speech 
recognition system may be linked to the text transcript generated by that system. 
Then, in the context of a word processing system, edits made to the transcript 
may be easily mapped back to this low level, internal information and used to 
automatically correct other, similar parts of the transcript and train the speech 
recognition system to improve future accuracy. This is entirely unrelated to the 
problem of generating collateral information queries that may be used to search 
external knowledge repositories, such as document collections, encyclopedias, 
fact files, almanacs, etc., and connecting the found collateral information back 
into the original input information stream. The technique described by Bunce 
teaches nothing about how to accomplish this. Thus, the combination of Kupiec 
and Bunce does not teach the combination of subsystems and database recited 
in claim 18. 

For the reason set forth above, it is submitted that the rejection of claim 18 
under 35 U.S.C 103(a) is inapplicable and should be withdrawn. 
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The Office Action rejects claims 3, 6, 8, 10, 1 1 , 26, 27, 31 and 32 
under 35 U.S.C 103(a) as unpatentable over Kupiec in view of Ford and further in 
view of Wactlar. 

Regarding claim 3, Kupiec makes no mention of the use of named entities 
(e.g., proper names, people, places, companies, etc.). A named entity is a 
sequence of one or more words that has been explicitly recognized as a specific 
person, place, company name, etc., and a corresponding semantic meaning has 
been attached to the named entity. Named entities are not mentioned anywhere 
in the entire text of Kupiec, and certainly not in column 2, lines 45-52 or column 
4, lines 66-67 as cited by the Examiner. Wactlar says nothing about the use of a 
taxonomy (the word is not to be found in the entire text of Wactlar), let alone the 
use of a taxonomy to calculate a taxonomy path score as recited in claim 3. 

Regarding claim 6, Kupiec at column 34, line 29, refers to searching for a 
word pair from the user's query in order to retrieve co-occurrence statistics and 
select the most likely word pair (hypothesis) from a set of candidate word pairs. 
This has nothing to do with automatically generating a query from an information 
stream based on a topic identified in the information stream. As noted above, 
Wactlar says nothing about the use of a taxonomy. 

Regarding claim 8, Kupiec only deals with explicit queries composed by 
the end user and does not teach how to automatically generate queries from an 
arbitrary input information stream. Wactlar describes using closed caption text 
for purposes of indexing video, a technique that is completely unrelated to the 
problem at hand of automatically generating queries from an arbitrary input 
stream in order to find collateral information. 
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Regarding claim 10, Wactlar does not address the problem at hand of 
automatically generating queries from an arbitrary input stream in order to find 
collateral information. The faces shown in Wactlar's Figs. A-1 and A-2 are not 
images contained in the information stream. 

Regarding claim 11, Wactlar says nothing about the use of a "taxonomy" 
or "named entities" (these concepts are not to be found anywhere in the text of 
Wactlar), and, as stated above, Wactlar does not address the problem solved by 
the claimed invention. 

Regarding claim 26, Wactlar describes generating a speech transcript 
from an audio/video stream for purposes of indexing the audio/video stream. 
However, neither Wactlar, Kupiec, nor Ford teach how to automatically generate 
queries from this or any other information stream for the purpose of searching 
other knowledge repositories to find collateral information. Thus, the combination 
of Wactlar, Kupiec and Ford do not teach the combination of steps recited in 
claim 26. 

Regarding claim 27, Wactlar does not teach how to analyze the television 
broadcast signal to automatically generate queries for the purpose of searching 
other knowledge repositories to find collateral information relevant to the 
television broadcast signal. 

Regarding claim 31, as described above, Kupiec at column 34, line 29, 
refers to searching for a word pair from the user's query in order to retrieve co- 
occurrence statistics and select the most likely word pair (hypothesis) from a set 
of candidate word pairs. This has nothing to do with automatically generating a 
query from an information stream based on a topic identified in the information 
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stream. Wactlar says nothing about the use of topic taxonomies, especially as 
recited in claim 31. 

Regarding claim 32, Wactlar says nothing about the use of topic 
taxonomies and automatically generating a query from an information stream 
based on a topic identified in the information stream. 

For the reason set forth above, it is submitted that the rejection of claims 
3, 6, 8, 10, 1 1, 26, 27, 31 and 32 under 35 U.S.C 103(a) is inapplicable and 
should be withdrawn. 

The Office Action rejects claims 19-23 under 35 U.S.C 103(a) as 
unpatentable over Kupiec in view of Bunce and further in view of Wactlar. . 

Regarding claim 19, Kupiec says nothing about named entities and their 
use in ranking search results. Wactlar says nothing about the use of topic 
taxonomies and automatically generating and scoring a query from an 
information stream based on a topic identified in the information stream. In 
column 13, lines 13-15, Wactlar does mention "classifying" motions of highly 
articulated objects, but this is a classification of visual object movement, and 
should not be confused with the use of a topic taxonomy for the classification of 
text (words, phrases, sentences) into one or more topics. 

Regarding claim 20, while the combination of Kupiec, Bunce and Wactlar 
might be used to automatically generate a digital library from the audio/video 
input stream, that is not what claim 20 recites. Claim 20 recites a system that 
automatically generates queries to find collateral information relevant to the input 
stream. As stated previously, this is not taught by either the combination of 
Kupiec and Bunce or the combination of Kupiec and Wactlar. Similarly, the 
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combination of Kupiec, Bunce and Wactlar does not teach the invention claimed 
by claim 20. 

Regarding claims 21 and 22, the combination of Kupiec, Bunce, and 
Wactlar does not teach automatically generating queries from the text extracting 
unit for the purpose of finding collateral information relevant to the information 
stream, regardless of how the text is extracted from the arbitrary input stream. 

Regarding claim 23, Wactlar says nothing about named entities, 
taxonomies, automatically generating queries from arbitrary information streams 
to find collateral information relevant to the information stream, and ranking the 
collateral information based on said plurality of criteria. 

For the reason set forth above, it is submitted that the rejection of claims 
19-23 under 35 U.S.C 103(a) is inapplicable and should be withdrawn. 

The Office Action rejects claims 28-30 under 35 U.S.C 103(a) as 
unpatentable over Kupiec in view of Ford and further in view of U.S. Patent No. 
5,786,814 to Moran et al., hereafter Moran. 

Regarding claim 28, neither Kupiec, Ford, nor Moran teach a system that 
addresses the problem of analyzing the information stream to automatically 
generate queries, search relevant databases, and find collateral information 
relevant to the current information stream. Moran, in the context of meetings, 
has a goal to provide the meeting participants with additional information relevant 
to the current topic of discussion. Moran only addresses capturing the meeting 
for archiving and later search and review. Accordingly, the combination of 
Kupiec, Ford, and Moran does not address the problem solved by the method 
recited in claim 28. 



19 



Regarding claim 29, as in claim 28, Moran only addresses capturing the 
meeting for archiving. Moran says nothing about analyzing meeting content as it 
is generated during the meeting for purposes of providing meeting participants 
with relevant, collateral information while the meeting takes place. However, the 
additional or collateral material is not disclosed as being obtained from a 
database based on the captured meeting content. 

Regarding claim 30, Moran only addresses archiving the actual meeting 
content. Since Moran does not teach how to automatically find collateral 
information during the meeting and make it available to meeting participants, 
Moran does not teach how to archive the collateral information. 

For the reason set forth above, it is submitted that the rejection of claims 
28-30 under 35 U.S.C 103(a) is inapplicable and should be withdrawn. 

The Office Action rejects claim 33 under 35 U.S.C 103(a) as unpatentable 
over Kupiec. 

This rejection is untenable for the reasons set forth above in the 
discussion of the rejections of claims 1, 12, 18 and 24. Namely, Kupiec does not 
teach analyzing an arbitrary information stream to recognize a presence of 
events that occur therein, automatically generate database queries from the 
recognized events, analyzing the database search results so as to rank and 
select the results for insertion into the information stream as collateral 
information. Kupiec only deals with explicit queries composed by the end user. 
Accordingly, Kupiec does not teach a computer readable media having recorded 
thereon a program to perform the claimed operations of the program. 

For the reasons set forth above, it is submitted that the rejection of claim 
33 under 35 U.S.C. 103(a) is erroneous and should be withdrawn. 
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It is respectfully requested for the reasons set forth above that the 
rejections under 35 U.S.C. 103(a) be withdrawn, that claims 1-33 be allowed and 
that this application be passed to issue. 
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